Likelihood and Smoothing Spline Analysis of Variance

We discuss a class of methods for the problem of `soft' classi cation in supervised learning. In `hard' classi cation, it is assumed that any two examples with the same attribute vector will always be in the same class, (or have the same outcome), whereas in `soft' classi cation, two examples with the same attribute vector do not necessarily have the same outcome, but the probability of a particular outcome does depend on the attribute vector. In this paper we will describe a family of methods which are well suited for the estimation of this probability. The method we describe will produce, for any value in a (reasonable) region of the attribute space, an estimate of the probability that the next example will be in class 1. Underlying these methods is an assumption that this probability varies in a smooth way (to be de ned) as the predictor variables vary. The method combines results from Penalized log likelihood estimation, Smoothing splines, and Analysis of variance to get the PSA class of methods. In the process of describing PSA we discuss some issues concerning the computation of degrees of freedom for signal, which has wider rami cations for the minimization of generalization error in machine learning. As an illustration we apply the method to the Pima-Indian Diabetes data set in the UCI Repository, and compare the results to Smith et al(1988) who used the ADAP learning algorithm on this same data set to forecast the onset of diabetes mellitus. If the probabilities we obtain are thresholded to make a hard classi cation to compare with the hard classi cation of Smith et al(1988), the results are very similar, however, the intermediate probabilities that we obtain provide useful and interpretable information on how the risk of diabetes varies with some of the risk factors.